Light-Field Imaging 101

A comprehensive introduction to unfocused light-field imaging (plenoptic 1.0)

Intended Audience

This guide aims to expand upon existing unfocused light-field (plenoptic 1.0) imaging learning material by providing detailed Python implementations of light-field image processing along with examples. The light-field image processing workflow closely follows that of the MATLAB-based Light-Field Imaging Toolkit [1], but leverages Python’s open-source ecosystem and Quarto’s enhanced capabilities for visualization and LaTeX formula integration.

By focusing specifically on the image processing aspects, this guide helps readers better understand the technical limitations and practical considerations of light-field imaging techniques. This guide assumes the reader is familiar with unfocused light-field imaging. Such an understanding can be achieved by browsing plenoptic.info or reading the unfocused light-field part of the Light-Field Camera Working Principles chapter (Pages 11-25) of the book Development and Application of Light-Field Cameras in Fluid Measurements [2].

Preprocessing

Raw Images

To illustrate light-field image processing concepts, a synthetic unfocused light-field image was generated and is shown in Figure 1. The image describes the raw pixel intensities as recorded by an unfocused light-field camera. Every lenslet of the camera projects the incident radiance into a defocused intensity distribution pattern. This manifests as a spatially constrained blur where the light energy from each microlens is distributed across multiple sensor pixels rather than being concentrated at the expected conjugate position. The image was generated using a commercial ray tracing software (OpticStudio, ANSYS).

Figure 1: Synthetic unfocused light-field image. The hexagonal patterning in the image is characteristic of the projection process by the microlens array.

Microlens Array Structure

The unfocused light-field camera model implemented in this investigation utilizes a hexagonal microlens array configuration for two primary reasons. First, hexagonal microlens arrays provide optimal spatial packing efficiency, thereby maximizing the effective sensor area utilization. Second, the hexagonal pattern introduces additional computational complexity in light-field image processing algorithms—specifically in sampling pattern interpolation and spatial frequency analysis—that merits thorough examination within this methodological framework.

The hexagonal microlens array has 51 × 51 complete lenslets. This core array is supplemented with partial lenslets along the periphery to achieve an overall rectangular shape. The total number of elements in the array is 2 729, which corresponds to the expected number of spots in the calibration image.

Calibration

The calibration process seeks to pinpoint the centroid of each lenslet’s corresponding sensor region. A calibration image is acquired with the main lens’s aperture reduced to a minimum. The resulting calibration image consist of an array of small bright spots as shown in Figure 2.

Figure 2: Synthetic light-field calibration image. Every bright spot is indicative of the corresponding lenslet centroid on the sensor.

The centroid identification process based on intensity peaks in an image is a standard image processing technique widely documented in the literature, not exclusive to light-field imaging. Since the synthetic calibration image contains no sensor noise, the calibration procedure has been intentionally simplified to emphasize conceptual clarity and streamline the explanation.

The spot centroid detection procedure consists of three main steps. First, a manual threshold is applied to the calibration image to identify potential spot locations. Second, a binary dilation operation using a disk-shaped structuring element is performed to expand these areas, ensuring that the subsequent weighted centroid calculation encompasses the full spot and its immediate surroundings. Finally, the scikit-image regionprops function is used to calculate the intensity-weighted centroids of each spot.

from skimage import morphology
from skimage.morphology import disk
from skimage.measure import label, regionprops

# Load calibration image
calibration_image = tifffile.imread('calibration.tif')

# Apply hard-coded threshold (chosen to match the expected number of centroids)
binary_calibration = calibration_image > 16

# Perform a binary dilation to ensure coverage of the spots for the computation
# of the weighted centroid
dilated_calibration = morphology.binary_dilation(binary_calibration, disk(4))

# Create labels 
labeled_calibration = label(dilated_calibration)

# Apply regionprops
regions_calibration = regionprops(labeled_calibration, intensity_image=calibration_image)

# Compute weighted centroids location
centroids = [region.centroid_weighted for region in regions_calibration]

print('Number of spots detected:', len(centroids))

Number of spots detected: 2729

References

[1]

J. Bolan, E. Hall, C. Clifford, and B. Thurow, “Light-field imaging toolkit,” SoftwareX, vol. 5, pp. 101–106, 2016.

[2]

S. Shi and T. New, Development and application of light-field cameras in fluid measurements. Springer, 2023.